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FOREWORD 


Both  the  Deputy  Chief  of  Staff  for  Personnel  (DCSPER)  and  the  Train¬ 
ing  fuid  Development  Command  (TRADOC)  have  expressed  a  strong  need  to 
examine  the  training  and  personnel  requirements  of  weapon  systems  under 
development.  To  reduce  training  costs,  the  assessment  of  training  im¬ 
pacts  must  begin  as  early  in  system  development  as  possible.  OMB  Cir¬ 
cular  A109,  Major  Systems  Acquisition,  has  set  early  estimation  as  a 
major  policy  goal.  In  order  to  aid  the  Army  in  meeting  these  needs, 
the  ARI  Field  Unit  at  Fort  Bliss  has  begun  a  research  program  aimed  at 
the  development  of  an  early  training  assessment  technology.  The  problem 
is  a  complex  one  touching  many  areas  of  man/machine  interactions. 

The  research  reported  here  considers  a  broad  spectrum  of  training 
derivable  at  the  earliest  stages  of  weapon  system  specification.  An 
examination  of  the  state  of  the  art  is  made,  with  recommendations  for 
six  areas  of  training  research:  concept  generation,  task  specification, 
trade-off  analysis,  management  information,  system  effectiveness  esti¬ 
mation,  and  costing.  Innovative  and  little  known  techniques  include 
both  tri-service  and  foreign  research.  A  proposal  is  made  for  combina¬ 
tions  and  extensions  of  existing  research  to  meet  projected  Army  needs. 
Areas  in  need  of  further  research  are  identified. 

This  research  is  in  response  to  requirements  of  Army  Project 
2Q762722A777  and  special  needs  of  the  Directorate  of  Combat  Develop¬ 
ments,  Fort  Bliss,  Tex. 
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EARLY  TRAINING  ASSESSMENT  WITHIN  DEVELOPING  SYSTEM  CONCEPTS 


BRIEF 


Requirement: 

To  investigate  estimation  of  training  requirements  for  weapon  sys¬ 
tems  in  early  conceptual  development. 

To  propose  a  methodological  framework  for  estimation  of  training 
that  includes  existing  and  developing  research. 


Procedure : 

System  development  impacts,  threats,  hardware,  and  training  needs 
were  outlined  for  developing  systems.  Six  methodological  areas  were 
evaluated:  concept  generation,  task  specification,  trade-off  analysis, 
management  information,  effectiveness  estimation,  and  costing.  Strengths 
and  weaknesses  were  considered  for  each  area.  Ways  of  merging  existing 
techniques  were  elaborated.  Future  research  areas  were  identified. 


Findings : 

A  workable  structure  in  which  an  early  training  estimation  pro¬ 
cedure  could  be  developed  was  generated. 


Utilization: 

This  report  can  be  used  to  help  specify  requirements  which  should 
be  met  by  a  procedure  for  estimating  training.  Background  material  draws 
on  tri-service  as  well  as  foreign  efforts  with  detailed  cross  referencing 
to  current  Army  regulations. 

It  is  concluded  that  an  early  training  assessment  system  is  within 
the  range  of  developing  technologies. 
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EARLY  TRAINING  ASSESSMENT  WITHIN  DEVELOPING 
SYSTEM  CONCEPTS 


INTRODUCTION 


System  Development  Impacts 

The  growth  of  complex  military  systems  has  created  a  corresponding 
need  to  improve  estimation  of  system  concept  impacts  and  costs.  During 
the  mid-1960s,  the  Army  recognized  that  serious  shortfalls  existed  due 
to  the  fact  that  trainer  and  user  requirements  had  little  influence  on 
the  early  development  and  fielding  of  most  materiel  systems  (Gross, 
1977;  Knauer,  1977;  Krebs,  1977;  Taylor,  1975).  To  remedy  that  situa¬ 
tion,  in  October  of  1968  the  Army  published  DA  Pamphlet  11-25,  Life 
Cycle  Management  Model  for  Army  Systems.  Revised  in  1975,  this  119 
event  flowchart  for  system  development  has  three  major  areas  of 
emphasis : 

1.  The  development  and  acquisition  procedures  for  Army  Systems 
beginning  with  concept  investigation  and  continuing  through 
disposal  of  obsolete  equipment, 

2.  Generation  of  a  framework  within  which  supporting  models  and 
publications  can  be  developed,  and 

3.  The  creation  of  a  management  structure  for  coordination  of 
combat  development,  R&D  production  and  logistics  support, 
personnel  requirements,  training,  and  maintenance  of  weapon 
systems . 

In  this  complex  document,  each  developing  system  is  placed  into  one  of 
four  major  categories; 

1.  Conceptual  where  concept  development  and  prototype  generation 
take  place , 

2.  Validation  where  verification  of  preliminary  concepts  takes 
place  and  trade-offs  and  operational  tests  begin, 

3.  Full  scale  development  where  the  system  is  completely  engi¬ 
neered,  fabricated,  tested,  and  integrated  with  human  require¬ 
ments  including  personnel,  documentation,  doctrine,  and  or¬ 
ganization  ,  and 

4.  Production  and  deployment  where  operational  units  are  trained, 
equipment  is  provided,  and  logistics  support  is  established. 


1 


In  May  1978  at  the  "Atlanta  V"  conference  on  Systems  Acquisition 
Perspectives  (1978),  General  Robert  J.  Baer,  Deputy  Commanding  General 
for  Materiel  Development,  noted  that  recent  Army  documentation  such  as 
OMB  Circular  A-109  (1976)  forces  a  concentration  on  the  front  end  of 
the  acquisition  cycle  and  requires  frequent  reevaluation  of  changing 
military  threats.  He  stated  that  there  is  a  growing  need  to  obtain 
improved  estimation  methods  for  developing  systems  to  reduce  the  life 
cycle  period  from  the  current  17-20  years. 

Part  of  the  driving  force  in  General  Baer's  focus  on  early  stages 
of  system  development  was  that  human  factors  considerations  are  most 
cost  effective  if  they  can  be  identified  before  hardware  development 
restrains  the  freedom  of  potential  training  systems.  After  physical 
dimensions  and  characteristics  have  been  set,  a  sequence  of  events  is 
put  in  motion  that  ripples  throughout  a  weapon  system  and  results  i.' 
marked  increases  in  the  cost  of  changes.  Figure  1  illustrates  one  way 
in  which  the  flow  of  developmental  events  may  be  visualized. 


Threats 

The  initial  driver  of  system  development  begins  with  a  perceived 
enemy  threat  and  a  derived  hardware  concept  and  mission  to  overcome 
that  threat.  The  basic  Army  regulation  dealing  with  threat  analysis 
is  AR  381-11.  Threats  are  broken  into  three  periods:  short-range 
(0-2  years) ,  mid-range  (2-10  years) ,  and  long-range  (10  years  and  be¬ 
yond)  .  Threat  analysis  has  five  stated  purposes: 

1.  To  provide  an  assessment  of  foreign  capabilities  in  terms  of 
combat  materiel,  employment  doctrine,  environment,  and  force 
structure ; 

2.  To  provide  an  assessment  of  the  level  of  development  which 
the  economy,  technology,  and  military  forces  of  a  country 
have  or  could  attain; 

3.  To  affect  U.S.  planning  or  development  by  extending  or  by 
supplementing  available  intelligence  estimates  or  validated 
force  deficiencies; 

4.  To  provide  a  statement  of  a  threat  as  it  relates  to  a  specific 
U.S.  research  or  combat  developments  project;  and 

5.  To  fill  gaps  where  data  are  lacking  or  evidence  is  too  incon¬ 
clusive  to  permit  an  intelligence  estimate. 

Within  the  Department  of  the  Army,  the  Office  of  the  Assistant 
Chief  of  Staff  for  Intelligence  (OACSI)  provides  threat  products  for 
dissemination.  For  systems  under  development  validated  threat  scenarios 
are  required.  (These  are  described  in  greater  detail  in  AR  10-5.) 
Generation  of  the  threat  details  is  treated  as  an  intelligence  function. 


Figure  1.  Impacts  of  early  development  areas  affecting  training 


However,  the  assessment  of  the  Impact  of  a  given  threat  on  Army  plans, 
studies,  projects,  and  systems  is  not.  The  main  two  documents  related 
to  early  developmental  studies  are  AR  5-5  (study  plan  guidance)  and 
AR  70-27  (the  threat  interface  development  plan) . 

Based  upon  a  threat  and  the  current  force  deficiency,  possible 
hardware  concepts  are  defined.  These  initial  concepts  drive  the  later 
life  cycle  events  which  include  training  related  issues.  Materiel 
concept  investigation  is  generally  initiated  through  one  of  two  sources. 
First,  a  materiel  developer  may  achieve  a  significant  advance  in  tech¬ 
nical  capability  and  knowledge.  Second,  the  combat  developer  may  obtain 
a  validated  capability  goal.  An  example  is  a  goal  established  at  HQDA 
to  counter  a  validated  threat,  to  correct  an  operational  inadequacy  in 
existing  materiel,  to  reduce  high  consumption  of  resources,  or  to  ex¬ 
ploit  a  technological  breakthrough.  Capability  goals  are  in  turn  de¬ 
rived  from  a  variety  of  sources  such  as  national  policy  guidance.  Army 
readiness  postures,  simulations,  studies,  war  games,  or  other  analytic 
research.  This  process  is  considered  in  detail  in  AR  71-9,  AR  70-1, 
and  AR  1000-1.  For  this  paper.  Figure  1  sufficiently  illustrates  how 
a  developing  equipment  concept  begins  to  effect  system  training. 

The  operation  of  a  new  system  is  driven  by  the  physical  charac¬ 
teristics  of  the  system  hardware.  Hardware  characteristics  such  as 
reliability,  physical  dimensions,  and  resource  needs  impact  directly 
on  the  human  users  through  operator  and  maintenance  requirements.  These 
requirements  are  reflected  in  the  tasks  which  must  be  performed.  The 
tasks  combine  with  available  personnel  capabilities  to  determine  the 
training  approach  required.  The  approach  flows  from  the  need  to  meet 
performance  objectives  generated  by  the  threat  and  mission.  Performance 
objectives  in  turn  drive  a  course  structure  and  a  support  structure. 

The  course  structure  includes  the  method  and  content  for  training  the 
specific  personnel.  The  support  includes  the  facilities,  associated 
staff,  and  media. 


Hardware 

The  flow  of  system  hardware  development  therefore  leads  to  train¬ 
ing  and  personnel  impacts.  In  terms  of  the  Life  Cycle  Management  Model 
described  in  AR  11-25,  the  early  training  system  impacts  are  two  events 
in  AR  11-25;  number  4  (training  plans)  and  number  5  (organizational 
and  operational  concepts) .  Training  plans  are  developed  by  the  desig¬ 
nated  training  group  in  coordination  with  the  materiel  developer.  Com¬ 
bat  Developer  and  Logistician.  Details  of  this  process  are  found  in 
AR  71-5  and  AR  611-1.  In  general,  events  4  and  5  interact  with  the 
materiel  system  in  two  ways.  First,  training  demands  cam  be  produced 
by  changes  in  the  design.  Second,  a  strategy  chosen  for  training  on  a 
particular  design  concept  can  require  that  skilled  personnel  must  be 
available  when  the  system  is  deployed.  As  stated  in  AR  11-25: 


Training  plan  action  during  the  conceptual  and  validation 
phase  is  oriented  toward  the  establishment  of  training  con¬ 
siderations  which  will  influence  the  design  of  equipment 
and  identify  training  implications  that  have  an  impact  on 
material  readiness,  capability  and  overall  cost.  Planning 
during  development  centers  on  analysis  and  evaluation  of 
alternative  training  concepts.  The  training  plan  identi¬ 
fies  critical  areas  and  contributes  to  availability  and 
maintainability  objectives  as  well  as  requirements  for  in¬ 
clusion  in  the  Outline  Development  Plan  and  the  Development 
Plan.  Planning  includes  all  methods,  media,  devices,  skill 
qualification  tests,  training  extension  courses,  and  simu¬ 
lations  required  for  institutions  and  units. 

Organizational  and  operational  concepts  are  the  responsibility  of 
the  Combat  and  Materiel  Developers.  The  emphasis  is  on  organizational, 
equipment,  and  personnel  trade-offs  that  would  be  required  if  the  sys¬ 
tem  concept  is  included  in  the  total  force  structure.  The  result  of 
trade-off  studies  serves  as  the  basis  for  the  Provisional  Qualitative 
and  Quantitative  Personnel  Requirements  Inventory  (PQQPRI) ,  Doctrinal 
and  Organizational  Test  Support  Package,  and  Basis  of  Issue  Plan. 

Greater  detail  on  these  areas  may  be  found  in  AR  1-1,  71-9,  71-2, 

570-2,  and  750-1. 

Although  the  Life  Cycle  model  includes  the  entire  system  cycle, 
this  paper  examines  only  the  research  implications  of  AR  11-25  events 
4  and  5.  It  is  evident  that  many  of  the  training  aids  now  being  de¬ 
veloped  for  validation  and  later  stages  are  directly  effected  by  the 
quality  of  projections  performed  at  early  stages.  As  a  reflection  of 
this  relationship,  efforts  are  ongoing  to  improve  the  data  bases.  These 
efforts  include  the  TDIS  (Training  Developments  Information  System) 
being  set  up  at  Fort  Eustis,  Va.,  and  the  LSAR  (Logistics  Support  Analy¬ 
sis  Record)  at  DARCOM  and  CODAP  (Comprehensive  Occupational  Data  Analy¬ 
sis  Programs).  However,  the  implications  of  early  inclusion  of  training 
and  human  factors  considerations  go  far  beyond  data  bases.  As  illus¬ 
trated  in  Figure  1,  components  are  interrelated,  so  changes  in  one  area 
produce  impacts  in  many  others.  This  has  direct  implications  for  the 
needs  which  future  R&D  must  address. 


Training 

Figure  1  showed  a  general  overview  of  training  flows.  If  attention 
is  limited  to  the  training-related  breakout  of  the  implied  operations, 
Figure  2  can  be  derived.  Based  on  threat  and  current  operational  capa¬ 
bility,  a  deficiency  is  identified.  The  deficiency  leads  to  a  system 
need  concept  which  includes  materiel,  support,  and  doctrinal  and  organi¬ 
zational  capabilities.  From  these  capabilities  a  system  description  is 
derived  including  the  human  functions  which  must  be  performed.  Figure  2 
traces  the  utilization  of  these  functions  as  they  impact  training  devel¬ 
opment.  Statements  about  system  purpose  can  be  broken  into  two  parts: 
general  operations  which  represent  global  system  capabilities  and 


Figure  2.  Breakdown  of  conceptual  operations 


specific  operations  which  are  related  to  the  mission  scenario  within 
which  the  system  will  be  utilized.  General  operations  can  be  broken 
into  three  groups:  machine  only,  mixed,  and  human  only.  When  a  mis¬ 
sion  is  considered,  it  is  possible  to  identify  critical  operations  in 
terms  of  their  effect  on  the  threat.  The  interface  between  the  devel¬ 
oping  hardware  and  the  operational  groupings  become''  the  first  place 
where  improved  assessment  techniques  resulting  f '  R&D  could  have 

major  impact.  Pending  a  trade-off  result  accorc  j  to  event  5,  the 
output  of  this  interface  feeds  more  detailed  training  analysis. 

The  global  operations  are  then  redefined  in  terms  of  the  specific 
activities  required  for  their  performance.  The  activities  lead  to 
human  behaviors  which  can  be  broken  into  skills  and  knowledges.  One 
way  this  breakdown  could  occur  is  presented  in  Figure  3.  Activities 
are  separated  into  training  requirements  based  on  required  skills, 
knowledges,  and  projected  personnel  characteristics.  Based  on  mission 
critical  tasks,  a  training  mode  analysis  takes  place  which  sorts  activi 
ties  into  equipment  intensive  or  personnel  intensive  training.  The 
former  refers  to  training  devices  such  as  embedded  training  or  simu¬ 
lator  generation.  The  latter  refers  to  training  procedures  sensitive 
to  interpersonal  factors  such  as  role-playing  techniques  or  tutoring. 
Equipment  intensive  analysis  provides  a  second  point  at  which  training 
development  and  hardware  development  intertwine  and  where  R&D  efforts 
need  to  be  applied. 


SELECTED  METHODOLOGIES 

Overview 

Heretofore,  the  flow  of  information  during  concept  development 
has  been  considered  in  terms  of  impact  in  general  training  areas.  Now 
specific  research  efforts  will  be  addressed.  Conclusions  will  be  drawn 
regarding  what  remains  to  be  done  and  preliminary  suggestions  will  be 
made  for  completing  an  early  training  assessment  system  (ETAS) . 

The  focus  of  this  examination  will  be  on  processes  having  the 
greatest  impact  on  the  training  areas  in  Figures  2  and  3.  For  simpli¬ 
fication,  research  efforts  will  be  evaluated  under  the  following 
headings : 

1 .  Concept  generation , 

2.  Task  specification, 

3.  Trade-off  analysis, 

4.  Management  information, 

5.  Effectiveness  estimation,  and 

6 .  Costing . 

The  above  topics  cover  prohibitively  large  areas  of  research. 

It  would  go  beyond  the  purpose  of  this  review  to  attempt  a  discussion 
of  all  tri-service  research  efforts.  Instead,  a  small  number  of  high 
quality  studies  dealing  with  each  of  the  above  topics  have  been 


considered.  These  were  selected  to  integrate* into  a  comprehensive  set 
of  technologies  needed  during  early  conceptual  development  stages.  The 
benefit  of  such  an  approach  is  that  areas  still  lacking  research  become 
more  obvious  than  if  individual  topics  were  considered  piecemeal.  The 
disadvantage  is  that  the  research  screening  process  might  omit  signifi¬ 
cant  studies  whose  true  value  may  not  become  evident  until  they  are 
placed  in  the  light  of  an  overall  pattern.  It  is  hoped,  however,  that 
the  benefits  of  producing  the  overall  pattern  will  make  up  for  loss  of 
specific  elements. 

The  first  topic  to  be  considered  is  conceptual  generation.  Gen¬ 
eration  means  the  production  of  the  initial  system  concepts  based  on 
the  perceived  deficiencies  between  the  threat  and  the  current  operational 
capability.  As  illustrated  previously  in  Figure  1,  a  threat  leads  to 
an  examination  of  available  hardware  and  projected  future  technology 
which  can  be  applied  to  create  new  hardware  systems  capable  of  meeting 
a  mission  deficiency.  Physical  characteristics,  reliability,  and  logis¬ 
tics  needs  are  derived  from  a  hardware  concept.  From  the  standpoint  of 
the  psychologist,  the  question  to  be  asked  is  "how  does  the  concept  de¬ 
veloper  construct  the  first  estimate?"  Particularly,  how  does  the  docu¬ 
mentation  produce  impact  on  the  later  needs  of  a  training  analyst? 


Concept  Generation 

If  training  analysts  are  to  enter  into  the  development  process 
early,  they  must  be  able  to  communicate  with  the  hardware  designer  so 
the  needs  of  both  can  be  freely  exchanged.  This  in  turn  requires  a 
common  language  through  which  both  parties  can  communicate.  This  is 
true  not  only  for  the  interaction  between  training  and  hardware  devel¬ 
opment  but  also  between  analysts  of  each  group.  In  the  past  such  com¬ 
munication  has  not  always  taken  place.  This  is  often  due  to  the  fact 
that  technological  breakthroughs  may  be  known  only  to  limited  groups 
of  people  within  a  particular  scientific  area  or  industrial  firm.  A 
large  percentage  of  past  efforts  have  resulted  from  unsolicited  pro¬ 
posals  or  individual  interactions  between  the  military  and  civilian 
community.  The  exception  to  the  rule  is  when  a  perceived  threat  leads 
directly  to  the  generation  of  a  Mission  Elements  Needs  Statement  (MENS) 
or  Science  and  Technology  Objective  Guide  (STOG)  which  is  let  out  for 
potential  submissions.  A  problem  with  this  system  is  that  communica¬ 
tion  becomes  difficult  as  more  and  more  affected  parties  are  brought 
into  the  developmental  cycle.  Neglecting  specifics  of  how  an  initial 
design  is  produced,  the  first  problem  is  design  description  to  maximally 
integrate  the  input  needs  of  all  Army  personnel.  This  problem  encom¬ 
passes  a  large  range  of  psychological  research.  In  terms  of  training 
development,  a  driving  factor  is  that  descriptions  must  lend  themselves 
to  the  eventual  production  of  tasks  and  their  associated  learning  ob¬ 
jectives.  This  has  direct  implications  for  the  way  equipment  design 
must  take  place.  There  are  at  least  two  research  efforts  which  could 
be  used  to  improve  design  communication. 
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ECSL.  The  first  is  a  simulation  language  called  ECSL  (Extended 
Continuous  Simulation  Language)  (Clementson,  1978)  used  primarily  in 
Great  Britain.  ECSL  is  unique  in  several  respects.  First,  it  was  de¬ 
veloped  as  a  result  of  communication  problems  between  designers  and 
oil  company  executives  during  drilling  operations  in  the  North  Sea. 

It  became  evident  that  it  was  impossible  to  communicate  in  highly  tech¬ 
nical  hardware  terminology  to  nontechnical  personnel  responsible  for 
system  coordination.  To  remedy  the  problem,  a  common  graphic  mediator 
was  developed  by  which  each  user  could  make  his/her  understanding  of 
the  system  specific  by  means  of  a  graphic  analogy  called  activity  net¬ 
works.  Similar  to  scribbles  on  a  blackboard,  the  activity  networks 
broke  down  the  world  into  objects  waiting  to  perform  actions  and  the 
actions  themselves.  This  mediator  permitted  rapid  communication  of 
underlying  relationships  regardless  of  the  particular  terminology  which 
was  used  by  each  group.  To  study'  the  implications  of  a  given  system 
configuration,  ECSL  was  developed  as  FORTRAN-based  language  through 
which  activity  cycles  could  be  simulated.  Thus,  a  concept  could  be 
recorded  quickly  into  a  form  suitable  for  dynamic  exploratory  testing. 

CAPS.  The  researchers  did  not  stop  here  however.  The  next  step 
constituted  a  major  breakthrough  in  the  use  of  simulation.  Recognizing 
that  most  simulation  languages  such  as  ECSL  are  far  too  complex  for  the 
average  user,  a  user-transparent  machine  interface  called  CAPS  (Com¬ 
puter  Aided  Programming  System)  (Bailey,  Pash,  &  Watts)  was  created 
which  permitted  the  user  to  specify  the  activity  cycle  interactively 
with  the  computer  once  a  system  configuration  was  proposed.  Once  a 
network  was  produced  and  self-checked  by  the  computer  program  for  con¬ 
sistency,  the  CAPS  automatically  converted  the  model  to  ECSL  code  re¬ 
sulting  in  an  immediate  error  free  simulation!  The  implications  for 
training  development  are  enormous,  including  many  possible  uses  of 
rapid  simulation  in  military  problems.  Through  CAPS  the  training  de¬ 
veloper  and  the  hardware  developer  can  not  only  share  a  common  con¬ 
ceptual  language,  but  the  result  of  changes  can  be  immediately  evalu¬ 
ated  in  trade-off  simulations. 

CAPS  provides  a  well-developed  capability  for  improving  communi¬ 
cations  between  the  trainer  and  the  hardware  developer.  It  is  well 
documented  although  acquisition  would  still  require  foreign  contacts. 

In  terms  of  future  needs  CAPS  does  have  limits  as  a  result  of  the  ac¬ 
tivity  cycle  format  and  the  ECSL  base  language.  Tremendous  advances 
are  possible  if  the  same  concept  were  to  be  expanded  to  determine  an 
optimal  military  interface  language.  This  would  require  a  specialized 
human  interface  such  as  CAPS  for  communications  between  hardware  de¬ 
velopers  and  trainers.  The  Fort  Bliss  field  unit  is  presently  explor¬ 
ing  CAPS  as  well  as  research  requirements  for  optimal  simulation 
language  selections  as  part  of  an  ILIR  (Independent  Laboratory  In- 
House  Research)  effort.  Many  psychological  research  problems  remain; 
however,  this  relatively  unexplored  area  could  provide  great  benefits 
since  early  improvements  in  system  formulation  ripple  throughout  the 
life  cycle  at  ever  increasing  costs. 
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THOUGHTST I CKER .  A  second  interesting  effort  being  conducted  under 
an  Air  Force  Office  of  Scientific  Research  6.1  contract  is  called 
THOUGHTST ICKER.  THOUGHTST ICKER  is  a  computer-regulated  concept  inter¬ 
rogation  and  recording  device.  At  one  level  of  analysis,  this  system 
encourages  a  user  to  output  an  explanation  or  justification  of  each 
concept  and  then  checks  that  the  user  output  satisfies  rules  of  con¬ 
sistency  and  complete  cycles  of  logic.  A  completed  cycle  is  called 
an  entailment  mesh.  It  permits  the  use  of  graphic  analysis  techniques 
to  simplify  redundancies  and  helps  focus  a  designer  concept.  A  second 
user  mode  exists  which  permits  the  designer,  after  an  entailment  mesh 
has  been  produced,  to  defend  design  decisions  by  suggesting  generaliza¬ 
tions  which  analysts  may  accept  or  reject.  If  analysts  reject  these 
generalizations,  they  must  supply  a  logical  reason,  forcing  considera¬ 
tion  of  alternatives.  In  overview  THOUGHTSTICKER  is  interfacing  hard¬ 
ware  combined  with  computer  programs  and  consists  of  10  functional 
components : 

1.  A  mesh  display  on  which  the  user  writes  concept  derivation 
paths ; 

2.  A  graphic  display  tube  and  console  used  to  present  modified 
meshes,  as  well  as  to  elicit  descriptor  values  (Results  of 
over  generalizing  system  heuristics  such  as  "extrapolation 
of  principles"  are  output  to  users  as  proposals  through  this 
display.  The  current  system  uses  two  displays.); 

3.  A  terminal  input  device  for  user  control; 

4.  Files  for  entering  demonstration  materials  for  topics 
(Clusters  of  signal  lamps  referencing  each  file  have  dif¬ 
ferent  meanings  to  the  user  at  different  phases  in  the  pro¬ 
cess,  e.g. ,  to  indicate  the  status  of  a  topic  or  all 
simplifications.) ; 

5.  A  joystick,  used  to  point  out  topics  on  the  graphics  display 
tube  (For  example,  topics  for  which  files  are  to  be  updated.) 

6.  A  descriptor  display  also  used  to  present  indexed  items  or 
topics  which  are  subsequently  associated  with  other  topics  on 
request; 

7.  An  item  indexing  board  for  descriptor  elicitation  and  external 
regulation  of  the  system; 

8.  Computer  and  interface  hardware  (for  operating  peripherals) ; 

9.  A  display  for  exhibiting  photographs  of  previous  stages  in  the 
process;  and 

10.  A  board  for  retrieving  and  displaying  data  structures  of 
previous  stages. 
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In  a  typical  utilization,  a  designer  produces  an  entailment  mesh  under 
the  constrained  conditions  above.  He  then  modifies  the  mesh  as  a  re¬ 
sult  of  being  forced  to  examine  assumptions  and  generalizations.  Fi¬ 
nally,  the  system  requires  the  user  to  describe  the  mesh  by  citing 
descriptions  (actually  many  valued  sets  of  descriptive  variables)  as 
relevant  or  irrelevant.  Critical  conditions  such  as  design  relia¬ 
bility,  ease  of  understanding  weight,  cost,  size,  and  environmental 
sensitivity  are  required  as  mandatory  descriptors.  The  final  design 
mesh  leaves  a  standardized  form  for  use  by  others  much  as  the  activity 
cycles  provided  a  standard  input  for  CAPS. 

Future  Research  Needs.  THOUGHTSTICKER  is  a  developing  system 
used  primarily  in  the  context  of  electrical  engineering  although  it  .has 
potential  in  other  areas.  This  is  also  a  foreign  effort.  Its  greatest 
value  lies  in  illustrating  the  feasibility  of  creating  a  designers  aid 
which  can  automatically  standardize  a  description  within  which  system 
concepts  can  be  developed.  Research  is  needed  to  produce  an  advanced 
system  oriented  for  military  concept  development.  Research  must  also 
explore  the  relationships  between  network  structures  based  on  hardware 
or  conceptual  designs  and  the  resulting  impacts  on  networks  of  tasks. 

The  use  of  such  a  network  will  be  discussed  in  detail  in  a  later  section. 

During  early  system  development,  a  system  such  as  THOUGHTSTICKER 
could  provide  an  answer  to  a  variety  of  problems  associated  with  assess¬ 
ing  training  impacts.  The  ability  to  prestructure  conceptual  develop¬ 
ment  and  to  produce  a  transportable  output  could  provide  immediate 
benefits  via  wider  distribution  of  early  system  concepts.  System  ideas 
could  be  developed  more  easily  in  team  efforts  where  THOUGHTSTICKER  or 
a  CAPS  system  could  provide  standardization  of  output  forms  for  both 
equipment  and  task  derivations.  The  task  implications  of  a  particular 
hardware  concept  could  be  explored  as  an  integral  part  of  an  activity 
network  or  entailment  mesh.  Computerization  of  this  data  base  could 
permit  rapid  changes  or  updates  as  threat-driven  modifications  are  re¬ 
quired.  This  possibility  leads  to  a  requirement  for  determination  of 
what  information  should  be  included  in  a  task  base  derived  from  early 
conceptual  configurations.  These  needs  will  now  be  considered. 


Task  Specification 


A  much  larger  quantity  of  research  exists  for  task  specification; 
however,  many  of  the  efforts  are  not  system  oriented  to  the  extent 
that  subprocesses  can  be  broken  out  for  use  with  other  training  devel¬ 
opments.  Two  efforts  will  be  considered  which  do  hold  promise  however. 


Task  specification  actually  includes  several  subareas,  all  of 
which  ultimately  are  derived  from  implications  of  hardware  configura¬ 
tion  and  the  definition  of  expected  usage.  The  following  derivation 
can  be  made.  The  mission  plus  equipment  leads  to  patterns  of  usage. 
These  patterns  of  usage  can  be  broken  up  into  specific  behaviors.  The 
behaviors  in  turn  can  be  broken  down  based  on  mission-based  objectives 
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such  as  who  does  what  to  whom,  how,  when,  and  how  well.  Based  on  an 
estimated  profile  of  a  typical  human  user,  the  projected  deficiency 
between  the  user  skills  and  knowledges  and  the  behavioral  objectives 
lead  to  training  deficiencies.  Training  deficiencies  are  used  along 
with  system  objectives  to  select  training  methods,  strategies,  and 
constraints.  To  perform  such  a  selection,  objectives  must  be  evalu¬ 
ated  in  terms  of  constituent  tasks.  The  tasks  are  in  turn  analyzed 
through  various  relations  and  values  which  they  maintain  in  respect 
to  system  hardware. 

The  major  process  by  which  Army  training  objectives  are  currently 
being  produced  is  through  the  Army  Instructional  System  Development 
(ISD)  procedures  contained  in  the  TRADOC  350-30  series  of  documents. 
These  documents  provide  an  overview  of  the  training  development  process 
but  still  lack  much  detail  by  which  specific  training  development  ob¬ 
jectives  must  be  met.  In  terms  of  the  development  of  early  training 
assessment  it  becomes  important  to  find  analytic  procedures  which  can 
be  linked  directly  to  system  concepts. 

The  SAT  Program  for  the  B-l  Bomber.  A  most  comprehensive  effort 
in  training  system  development  is  contained  in  some  work  performed  by 
the  Calspan  Corporation  under  contract  to  the  Advanced  System  Division 
of  the  Air  Force  Human  Research  Laboratories  at  Wright  Patterson  Air 
Force  Base  (Ring,  Startz,  Gaidasz,  Menig) .  Major  aspects  of  this  work 
which  bear  directly  on  early  system  specification  will  be  discussed. 

The  Systems  Analysis  of  Training  (SAT)  approach  uses  two  sets  of  in¬ 
formation  as  a  starting  point.  The  first  is  a  catalog  of  display  and 
control  information  derived  from  a  specified  system  concept.  This 
computerized  catalog  includes  seven  pieces  of  information: 

1.  Control  and  display  names, 

2.  Synonyms  for  names  used  in  number  1, 

3.  Subsystem  identifiers  for  equipment  locations, 

4.  Physical  locations  of  the  controls, 

5.  Types  of  displays, 

6.  Values  or  range  which  the  displays  can  assume,  and 

7.  Additional  clarifying  comments. 

Based  on  the  display  catalog  equipment  and  a  mission  scenario,  tasks 
are  inferred  by  breaking  gross  classes  of  actions  into  four  levels  of 
detail.  In  descending  order  of  inclusiveness  they  are 

1.  Mission  segment  (such  as  reload  a  missile) , 

2.  Function  (such  as  drive  reloading  vehicle) , 

3.  Task  (such  as  start  reloading  vehicle),  and 

4.  Task  element  (such  as  turning  a  key) . 

The  lowest  category  (4)  is  subdivided  into  a  series  of  information 
strings  based  on  behavior  type,  timeline  or  sequence,  and  crew 
interactions. 
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There  are  nine  components  in  a  SAT  task  element  description. 

They  are 

1.  Title, 

2.  Number, 

3.  Person  performing, 

4.  Behavior  category, 

5.  Duration, 

6.  Crew  interaction, 

7.  Previous  task  elements, 

8.  Next  task  element,  and 

9.  Comments. 

The  latter  information  (9)  presents  an  extremely  valuable  source  of 
unclassified  factors  that  may  impact  the  program.  In  the  context  of 
the  earlier  discussion,  elements  1  through  9  present  the  first  estimate 
as  to  what  kind  of  information  should  be  considered  in  the  conceptual 
development  for  the  entailment  mesh  or  CAPS  simulations  discussed  earl¬ 
ier.  For  example,  it  should  be  possible  to  program  a  THOUGHTST ICKER- 
like  system  to  interrogate  and  check  a  development  team  for  all  the 
elements  in  both  tasks  and  equipment  tables  at  the  same  time.  This 
could  lead  to  the  production  of  a  first  pass  task  and  equipment  list 
for  examination  by  combat  developers,  training  developers,  or  required 
contractor  personnel  should  the  information  be  used  to  guide  the  gen¬ 
eration  of  a  detailed  statement  of  work.  An  example  of  the  behavioral 
elements  such  a  system  would  require  is  given  in  Figure  4.  A  sample 
format  for  task  information  also  taken  from  the  SAT  reports  is  presented 
in  Figure  5. 

Much  work  must  still  be  done  before  a  preliminary  assessment  of  a 
training  program  could  be  produced.  Based  on  a  format  such  as  that 
used  in  SAT,  task  information  must  still  be  broken  into  behavioral  ob¬ 
jectives.  The  SAT  procedure  accomplishes  this  as  follows.  First,  task 
element  data  are  partitioned  into  behavioral  components.  The  components 
are  labeled  as  either  skills  or  knowledges.  Skills  are  defined  as  ob¬ 
servable  actions  requiring  physical  coordination.  Knowledges  are  de¬ 
fined  as  covert  responses  with  five  levels: 

1.  Identification, 

2.  Recall, 

3.  Interpretation, 

4.  Calculation,  and 

5.  Prediction. 

Skills  and  knowledges  are  then  grouped  on  the  basis  of  categorical  com¬ 
monalities  into  behavioral  objectives.  A  behavioral  objective  is  con¬ 
sidered  to  have  11  parts  (see  Figure  2).  They  are 

1.  Title, 

2.  Initial  conditions  required, 

3.  Concurrent  behaviors. 


Figure  4.  Behavioral  objective  format  taken  from  SAT  reports. 


Example  of  task  element  behavior  taken  from  SAT  reports 


4.  Performance  criteria, 

5.  Other  competing  objectives  which  must  be  met, 

6.  Unusual  conditions, 

7.  Operators  used, 

8.  Interactions  required, 

9.  Task  elements  accounted  for  by  the  objective, 

10.  Criticality  to  mission,  and 

11.  Difficulty  to  perform. 

The  final  step  is  to  again  group  task  elements  into  processing  blocks 
on  the  basis  of  common  behavioral  objectives.  The  methods  by  which 
this  is  accomplished  are  not  as  well  developed  in  the  SAT  program. 

Future  Research  Needs.  In  its  current  form,  SAT  consists  of  a 
variety  of  computer-aided  data  base  programs  and  manual  coding  forms 
required  for  task  evaluations  and  requires  expert  support  or  evaluation. 
The  real  value  of  this  work  has,  in  the  opinion  of  the  author,  been 
overlooked  possibly  due  to  the  cancellation  of  the  B-l  program.  There 
is  a  great  deal  of  theoretical  and  applied  information  which  has  not 
been  considered  here  but  which  could  have  beneficial  application  in 
other  training  areas  such  as  Cost  and  Training  Effectiveness  Analysis 
(CTEA)  (TRADOC  Pamphlet  71-10,  1977).  Some  developmental  work  is  needed 
to  adapt  the  formats  to  Army  needs  and  interface  between  the  final  con¬ 
cept  form  and  the  SAT  logic  for  going  from  mission  to  behavioral  objec¬ 
tives.  Effort  would  be  required  in  order  to  determine  whether  expert 
intervention  would  still  be  required  based  on  the  state-of-the-art  in 
task  derivation.  Nevertheless,  the  SAT  work  represents  a  significant 
advance  in  many  of  the  problems  now  developing  in  proceduralized  train¬ 
ing  estimation.  The  SAT  work  is  somewhat  unique  in  that  a  comprehensive 
and  systematic  effort  was  made  to  carry  the  initial  logical  framework 
through  to  the  actual  generation  of  a  training  system  for  a  weapon  that 
was  still  under  early  development.  For  that  reason,  it  deserves  close 
examination. 


Trade-off  Analysis 

In  the  section  on  task  generation,  it  was  noted  that  expert  inter¬ 
vention  was  required  in  system  conceptual  development  as  well  as  speci¬ 
fication  of  tasks  and  behavioral  objectives.  An  important  area  directly 
related  to  intervention  includes  hardware  and  training  trade-offs.  These 
are  generated  by  dollar  costs  and  functional  requirements  inferred  from 
estimated  battlefield  effectiveness.  Trade-offs  can  be  considered  at 
two  levels.  The  first  deals  with  the  relationship  between  system  candi¬ 
dates  and  total  battlefield  posture.  This  level  is  considered  under 
the  Army  requirements  for  Cost  and  Operational  Effectiveness  Analysis 
(COEA)  (TRADOC  Pamphlet  11-8).  As  illustrated  in  Figure  1,  the  selection 
of  equipment  configurations  leads  to  the  later  requirements  for  human 
performance  capabilities  and  training  programs.  Training  suitability 
is  reflected  in  the  need  for  early  adjustment  of  potential  systems  in 
order  to  minimize  cost  impacts.  Training  assessment  has  been  given  the 


name  of  Cost  and  Training  Effectiveness  Analysis  or  CTEA.  At  one  level, 
CTEA  is  intended  to  provide  evaluative  information  of  cost  and  projected 
effectiveness  for  new  training  systems.  In  terms  of  input  to  COEA 
these  analyses  cam  result  in  the  acceptance  or  rejection  of  a  new  sys¬ 
tem  concept  during  early  development  if  it  can  be  shown  that  the 'system 
has  an  unacceptable  resource  or  cost  impact  on  training.  CTEA  has  high¬ 
lighted  the  Army  need  for  estimation  of  the  effect  of  potential  train¬ 
ing  systems  as  well  as  changes  within  already  existing  systems  or  sys¬ 
tem  concepts.  Since  trade-offs  for  both  training  amd  hardware  are  an 
inherent  characteristic  of  early  life  cycle  development,  it  is  important 
that  the  training  developer  has  tools  to  assess  the  impact  of  changes 
suggested  by  the  hardware  developer.  This  brings  the  question  back  to 
how  the  impact  of  alternate  system  components  should  be  assessed.  One 
way  already  suggested  is  to  use  the  output  of  CAPS  as  a  system  simula¬ 
tion.  While  this  may  work  in  a  very  early  COEA  context,  it  may  be  too 
crude  for  making  specific  training  decisions.  This  paper  will  not  ex¬ 
amine  research  which  addresses  training  decisions. 

LCCIM/TRAM.  The  first  technique  to  be  discussed  is  Life  Cycle 
Cost  Impact  Modeling  System  (LCCIM)  (Baran,  Czuchry,  &  Goclowski,  1978). 
LCCIM  was  part  of  an  Air  Force  effort  called  DAIS  (Digital  Avionics 
Information  System)  (Czuchry,  Doyal,  Frueh,  Baran,  &  Dieter ly,  1978) 
which  resulted  from  a  need  to  assess  potential  impact  of  avionics  in¬ 
tegration  on  weapon  system  life  cycle  cost  and  system  support  personnel 
requirements.  The  DAIS  effort  went  beyond  original  project  goals  to 
consider  both  system  design  and  modification  phases  (Digital  Avionics 
Information  System) .  The  developing  LCCIM  concept  consists  of  three 
submodels  and  associated  data  banks  which  operate  either  interactively 
or  independently.  The  first  is  a  reliability  and  maintainability  model 
vdiich  traces  support  maintenance  operations  at  the  unit,  subsystem,  or 
system  level  to  produce  point  estimates  of  human  resource  requirements. 
According  to  the  authors,  it  can  also  identify  sources  of  high  resource 
consumption  and  answer  "what  if"  questions  concerning  the  expected 
results  from  changing  values  of  reliability  and  maintainability 
parameters. 

The  second  submodel  is  a  system  cost  program  which  aggregates 
components  of  system  life  cycle  cost  and  presents  them  either  in  se¬ 
lective  combination  or  summary  form.  This  cost  process  already  has 
several  good  analogs  within  the  Army  which  could  probably  be  applied 
quickly  when  needed  without  having  to  adopt  the  accounting  schemes 
used  by  the  Air  Force  or  Navy  (Braby,  Henry,  Parrish,  &  Swope,  1975) 
programs . 

The  third  submodel  is  perhaps  the  most  interesting  from  the 
standpoint  of  the  psychologist  and  parallels  well-developed  research 
within  ARI  on  the  training  developers  decision  aid  (TDDA)  (Pieper, 

Guard,  Michael,  &  Kordek,  1978),  TRAINVICE  (Wheaton,  Rose,  Fingerman, 
Korotkin,  S  Holding,  1976),  and  the  Fort  Bliss  CTEA  selection  metho¬ 
dology  (Jorgensen  &  Hoffer,  1979).  TRAM  (Training  Analysis  Model) 
(Digital  Avionics  Information  System)  includes  three  modules: 
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a  preprocessor  and  two  analytical  modules  for  trailing  plan  and  train¬ 
ing  program  generation.  To  function,  the  model  requires  an  existing 
data  bank  containing  the  set  of  tasks  to  be  learned.  The  generation 
of  the  data  bank  is  not  detailed  as  in  the  SAT  effort  considered  earlier. 

The  generation  of  such  a  bank  for  the  Army  still  remains  a  prob¬ 
lem.  The  Army  does  have  several  ongoing  efforts  to  produce  information 
banks  for  developer  systems.  One  consists  of  the  application  of  CODAP 
Comprehensive  Occupational  Data  Analysis  Programs  developed  by  Phalen 
and  Christal  (1973) .  A  more  recent  effort  for  earlier  systems  is  the 
LSAR  or  Logistics  Support  Analysis  Record  being  developed  and  utilized 
by  DARCOM.  A  third  effort  still  in  preliminary  stages  is  an  attempt 
to  unify  task  information  in  a  single  data  bank  across  MOS's  for  Train¬ 
ing  Development  Information  System  (TDIS)  at  Fort  Eustis,  Va. 

Returning  to  the  TRAM  model,  the  task  data  banks  are  subject  to 
user-defined  specifications  which  allow  the  assignment  of  five  descrip¬ 
tor  values  denoting  frequency,  criticality,  learning  difficulty,  taxonomy, 
and  sequencing.  This  information  is  input  into  a  preprocessor  which 
screens  the  total  set  of  tasks  in  a  series  of  "go"  or  "no  go"  decisions 
to  select  those  to  be  used  for  training.  The  output  of  the  selection 
process  is  used  to  feed  another  module  which  is  a  training  plan  gener¬ 
ator.  Based  on  a  series  of  real  world  constraints  including  the  number 
of  personnel  required,  maximum  allowable  training  cost,  and  maximum 
allowable  time,  the  generator  produces  an  initial  training  plan  in 
which  a  School/OJT  mix  is  determined.  This  is  followed  by  recommenda¬ 
tions  concerning  appropriate  methods  and  media.  The  process  appears  to 
require  a  high  degree  of  manual  and  expert  intervention.  The  TDDA  model 
may  possess  a  better  mix  of  skill  requirements  for  Army  utilization  in 
this  area. 

TEEM  (Training  Efficiency  Estimation  Model) .  A  second  model  to  be 
considered  for  trade-off  analysis  is  the  ARI  CTEA  work  (Jorgensen  et 
al. ,  1978) .  Since  CTEA  is  inherently  oriented  toward  upper  level  de¬ 
cision  makers  some  form  of  trade-off  analysis  is  critical.  Currently, 
ongoing  efforts  at  Fort  Bliss  have  focused  on  two  areas.  The  first 
area  includes  major  cost  impacts.  ARI  is  using  a  modified  form  of  the 
Navy  Training  and  Evaluations  Groups  cost  program  (Braby  et  al. ,  1975) 
or  Training  Evaluation  Cost  Evaluation  Program  (TECEP) .  Although  being 
well  suited  to  overall  cost  analysis,  TECEP  does  possess  some  weak 
points  in  that  it  is  not  oriented  toward  an  average  cost  accountant  but 
rather  is  more  of  a  theoretical  model  of  economics.  Problems  are  cur¬ 
rently  being  resolved  by  the  Army  Air  Defense  School.  The  TECEP  model 
does  provide  a  good  starting  point  for  cost  estimations  at  early  stages 
in  system  development.  As  systems  become  more  developed,  a  more  gen¬ 
eral  life  cycle  cost  model  would  probably  have  to  be  used. 

The  second  focus  of  the  Fort  Bliss  work  involves  the  Training  Ef¬ 
ficiency  Estimation  Model  (TEEM)  research.  The  TEEM  (Jorgensen  et  al., 
1975)  is  the  result  of  in-house  efforts  to  determine  the  efficiency 
by  which  training  resources  are  heing  utilized  to  meet  psychological 
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requirements  during  early  training  program  media  and  method  selections. 
The  efficiency  metric  is  described  in  detail  in  Jorgensen  and  Hoffer 
(1978)  and  tracks  the  degree  of  fit  between  task  descriptions  and  train¬ 
ing  hardware  selections.  It  could  form  a  basis  for  a  trade-off  analysis 
procedure  in  early  system  estimation.  Some  contract  efforts  in  this 
direction  are  already  underway. 

MODIA.  A  third  work  which  should  be  considered  in  this  area  is 
the  MODIA  model  developed  by  the  Rand  Corporation  (Carpenter,  1978) . 
Method  of  Designing  Instruction  Alternatives  (MODIA)  has  as  its  pri¬ 
mary  purpose  the  improvement  of  training  resource  utilization.  MODIA 
has  four  components: 

1.  A  description  of  various  options  for  course  design, 

2.  A  user  interface  to  facilitate  use  of  noncomputer  oriented 
personnel , 

3.  A  resource  utilization  model,  and 

4.  A  cost  model. 

Expert  intervention  is  provided  at  two  points  in  MODIA:  the  user 
interface  and  the  cost  model.  The  description  of  options  for  actual 
course  design  is  in  a  separate  manual.  The  resource  utilization  model 
is  a  stand  alone  computer  mode.  Regarding  functions,  the  first  component 
introduces  the  user  to  the  required  data  for  the  model,  the  choice  op¬ 
tions  which  will  be  available  when  the  interactive  user  interface  is 
run,  and  the  pros  and  cons  of  the  various  training  choices  as  the  au¬ 
thors  see  them.  Component  two  produces  interactively  refined  course 
descriptions  that  include  content,  teaching  strategy,  student  charac¬ 
teristics,  and  physical  resource  assignments.  The  interactive  nature 
of  the  procedure  is  valuable  because  it  allows  the  users  to  see  the  im¬ 
pact  of  trade-off  decisions  they  have  made  at  each  stage  of  the  process. 
Component  three  simulates  student  progression  through  the'course  struc¬ 
ture  developed  in  stage  two  and  generates  impact  requirements  from  stu¬ 
dent  flow  patterns  over  time,  waiting  times  for  required  resources,  and 
demands  upon  existing  supplies  and  facilities.  The  fourth  component 
consists  of  a  cost  model  which  estimates  5-year  investment  and  operation 
costs . 

To  use  the  MODIA  system,  two  groups  of  individuals  are  required. 

The  first  is  a  small  team  highly  familiar  with  the  logic  and  operation. 
The  second  group  includes  subject  matter  experts  who  are  normally  used 
in  course  development.  Working  as  a  team,  these  two  groups  answer  a 
series  of  questions  asked  by  the  system  via  the  user  interface  program. 
There  are  eight  areas  which  are  included  in  these  questions.  They  are 
(a)  the  training  objective  list,  (b)  the  subject  matter  type  (such  as 
team,  individual,  or  classroom),  (c)  the  training  examination  charac¬ 
teristics  (such  as  failure  rate  expected) ,  (d)  the  student  population 
characteristics  (such  as  arrival  times,  group  size,  and  ability  level). 


(e)  the  teaching  policy  (including  the  number  of  course  tracks,  the 
content  diversity,  and  learning  events) ,  (f)  the  teaching  method, 

(g)  the  test  characteristics,  and  (h)  the  physical  resource  demands. 


Future  Research  Needs.  MODIA  appears  to  present  possible  sources 
of  training  information,  particularly  in  the  resource  utilization  model. 
In  terms  of  immediate  Army  usage,  many  of  the  variables  are  highly  de¬ 
pendent  on  the  subjective  abilities  of  the  training  analyst  who  is  us¬ 
ing  the  system.  MODIA' s  value  may  lie  more  in  the  area  of  program  man¬ 
agement  than  in  actual  course  development  and  could  be  beneficially 
applied  for  examination  of  resource  trade-offs.  An  examination  of  the 
user  interface  programming  logic  to  see  how  the  various  student  and 
personnel  variables  impact  the  overall  system  would  be  valuable.  This 
area  is  particularly  lacking  in  other  research.  The  variables  which 
have  proved  useful  for  the  resource  utilization  model  output  may  also 
provide  valuable  insights  in  this  area. 

Trade-off  Summary.  Each  of  the  approaches  considered  for  trade¬ 
off  analysis  has  strong  and  weak  points.  LCCIM  appears  to  be  the  most 
inclusive  model  incorporating  elements  of  the  SAT  approach  as  well  as 
certain  submodels  present  in  the  MODIA  work.  The  TEEM  is  mostly  focused 
on  the  trade-offs  taking  place  in  the  choice  of  training  media  and 
method  selection.  It  does  not  take  into  account  the  broad  picture  of 
management  variables  which  impact  on  the  overall  training  system.  MODIA 
tends  to  be  more  global  particularly  regarding  personnel.  The  Navy  DOTS 
effort  which  will  be  considered  next  is  also  strong  in  this  area. 

In  terms  of  early  training  system  assessment,  it  appears  that 
many  of  the  model  inputs  could  be  fed  by  a  SAT  front  end.  However, 
making  the  components  mutually  compatible  or  selecting  the  strongest 
parts  from  each  would  require  significant  programming  efforts.  Assist¬ 
ance  in  this  area  may  soon  be  provided  as  a  spinoff  of  a  new  CTEA  con¬ 
tract  at  Fort  Bliss  designed  to  integrate  existing  technologies  into 
the  overall  Life  Cycle  System  Management  Model.  However,  many  of  the 
efforts  are  so  specialized  that  they  may  exceed  the  capability  to  de¬ 
velop  a  data  base  in  the  earliest  system  development  stages.  Research 
is  needed  on  procedures  for  early  forecasting  of  personnel  resources. 
Identification  of  components  in  the  resource  impact  portions  of  the 
models  is  needed  as  well  as  the  effects  and  interactions  of  various 
training  resource  on  the  performance  of  the  students  who  will  be  going 
through  the  courses. 


Management  Information 

Many  of  the  decisions  directly  affecting  the  validity  of  training 
program  prediction  cure  dependent  on  managerial  variables  as  well  as 
psychological  requirements  of  training  program  optimization.  This  im¬ 
plies  that  early  program  prediction  should  include  a  means  for  includ¬ 
ing  projected  impacts  of  specific  management  decisions  as  soon  as  pos¬ 
sible.  Not  only  must  the  initial  formulation  of  equipment  and  training 


tasks  be  sent  to  the  hardware  developer  and  training  developer,  but 
they  should  also  interface  with  the  probable  future  system  managers. 

This  may  be  extremely  difficult  at  the  earliest  stages  since  many  de¬ 
cisions  cannot  be  anticipated  until  impacts  of  future  circumstances 
are  known.  Nonetheless,  a  logical  choice  for  early  managerial  inter¬ 
vention  is  a  Life  Cycle  System  Manager  such  as  the  TRADOC  System  Mana¬ 
ger  (TSM) . 

Pour  efforts  of  potential  value  for  early  management  decisions 
will  be  presented.  The  first  two,  the  TSM  Guide  and  STEPS  (Simulation 
and  Training  Equipment  Planning  Sources) ,  deal  with  the  structure  within 
which  management  impacts  are  generated.  The  second  two,  DOTS  and  SNAP, 
deal  with  aids  through  which  a  manager  could  introduce  decisions  into 
a  developing  system  concept. 

TSM  Guide  and  STEPS.  Because  of  the  Army's  emphasis  on  the  over¬ 
all  LCSMM  (DoD  Pamphlet  5001) ,  ARI  undertook  a  research  program  in  1976 
to  systematically  examine  the  overall  flow  of  information  requirements 
impacting  the  newly  created  TRADOC  system  manager.  Applied  Science 
Associates  under  contract  to  ARI  produced  a  guidebook  which  describes 
how  training  development  and  acquisition  activities  fit  into  the  LCSMM 
for  total  system  development.  The  guide  consists  of  four  sections. 

The  first  one  discusses  the  main  elements  of  the  training  subsystem 
requirements.  The  second  section  presents  a  generalized  training  de¬ 
velopments  model  based  on  the  Army  Instructional  System  Development 
(ISD)  approach.  The  third  section  outlines  the  LCSMM  in  a  framework 
oriented  toward  training  new  managers  and  includes  major  milestones 
and  events.  The  final  section  integrates  training  development  activi¬ 
ties  with  the  total  system  acquisition  process  and  sketches  the  role 
of  the  TRADOC  system  manager  for  the  conduct  and  coordination  of  these 
activities. 

The  guide  is  most  useful  for  a  system  overview;  however,  it  does 
not  present  the  manager  with  the  specific  sources  from  which  evaluative 
data  are  to  be  supplied.  In  the  context  of  the  research  already  dis¬ 
cussed,  this  need  would  be  analogous  to  trying  to  apply  the  MODlA-user 
interface  without  the  introductory  manual  which  gives  the  information 
sources  required  by  the  program.  The  ARI  STEPS  effort  now  in  progress 
is  designed  to  meet  this  need  by  drawing  together  various  agencies  and 
sources  to  supply  the  data  base  information.  At  the  time  of  writing, 
the  STEPS  effort  is  still  ongoing  but  should  be  completed  in  early  fis¬ 
cal  year  1979. 

Future  Research  Needs  TSM/STEPS.  Although  providing  general  in¬ 
formation,  neither  STEPS  nor  the  TSM  guide  are  focused  heavily  at  the 
early  end  of  system  development.  They  do  provide  an  important  overview 
of  how  an  early  prediction  system  must  later  integrate  into  the  overall 
flow  of  military  activities.  Thus,  they  are  also  important  for  the 
development  of  managerial  aids  used  in  the  concept  development  stages 
prior  to  milestone  one  of  the  LCSMM.  Future  research  is  still  required 
to  determine  the  potential  impacts  of  an  improved  front  and  concept 
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development  system  on  training  system  development.  This  must  wait  on 
a  more  precise  determination  of  the  form  which  such  a  system  will  ul¬ 
timately  take. 

SNAP  and  DOTS.  Once  a  manager  has  obtained  an  overview  of  system 
impacts  and  potential  sources  of  data,  it  is  still  necessary  to  obtain 
inforo&tion  from  predictive  aids  in  order  to  make  choices  which  would 
impact  on  system  development.  In  a  global  sense,  such  management  aids 
are  already  under  consideration  in  the  development  of  COEA  and  CTEA 
techniques.  For  early  system  development,  however,  there  exists  a  need 
for  specific  management  tools.  Since  it  is  still  premature  to  predict 
what  degree  of  management  intervention  might  be  required,  two  such 
tools  will  be  considered,  each  at  opposite  ends  of  the  scale  in  terms 
of  technical  sophistication.  The  first  tool  is  a  graphic  decision  aid 
called  SNAP  (Simplified  Network  Analysis  Portrayal)  (Brown,  1977) . 

Many  manager  aids  have  been  produced  for  complex  programs.  Among  the 
better  known  are  simple  time-line  GANTT  charts  and  sophisticated  PERT/ 
CPM  techniques.  For  conceptual  development  the  former  is  probably  much 
too  simple  because  options  are  not  available  in  GANNT  charts  for  con¬ 
sideration  of  alternative  courses  of  action.  The  PERT  techniques  are 
powerful;  however,  they  require  an  extensive  effort  for  a  busy  manager. 
Consequently,  two  approaches  seem  reasonable  for  early  conceptual  sys¬ 
tems.  The  first  approach  is  the  use  of  a  graphic  aid  that  can  express 
problem  complexity  but  does  not  require  special  resources;  the  second 
is  the  use  of  a  sophisticated  system  by  creating  a  transparent  user 
interface  so  the  program  complexity  is  masked  behind  an  interactive 
format  much  like  that  used  by  MODIA  or  CAPS.  To  be  of  value  either 
approach  should  address  at  least  six  management  factors: 

1.  The  method  should  be  sensitive  to  the  level  of  complexity 
of  the  required  program  structure. 

2.  The  math  or  analytic  support  required  should  be  derivable  in 
advance . 

3.  The  complexity  of  the  rationale  behind  decisions  should  be 
evident . 

4.  The  output  information  should  be  explicit. 

5.  The  applicability  of  the  work  to  other  areas  should  be 
derivable. 

6.  The  resources  required  to  implement  an  action  should  be 
considered . 

SNAP .  SNAP  is  a  graphic  flow  charting  procedure  specifically 
oriented  toward  system  development  managers  (Brown,  1977).  It  was  de¬ 
veloped  as  part  of  a  project  for  the  Defense  Systems  Management  College 
at  Fort  Belvoir,  Va.  SNAP  consists  of  a  series  of  rules  for  flowchart¬ 
ing  managerial  decisions.  Events  in  system  management  are  characterized 


in  terms  of  strings  of  actions,  followed  by  evaluations,  followed  by 
decisions.  Decisions  in  turn  branch  to  a  new  series  of  actions.  The 
value  of  the  procedure  is  that  it  serves  as  a  graphic  representation 
of  perceived  impacts  resulting  from  various  management  decisions.  A 
graphic  framework  permits  not  only  the  decisionmakers  but  also  the 
system  developers  to  trace  projected  impacts.  The  disadvantage  of 
the  procedure  is  that  it  is  heavily  dependent  upon  the  skill  of  a 
manager  in  identifying  critical  decisions.  Often  in  a  complex  system, 
important  interactions  may  not  become  evident  until  the  entire  system 
functions  together.  Thus  gain  from  the  use  of  a  simple  graphic  pro¬ 
cedure  may  be  outweighed  by  the  importance  of  global  effects. 

DOTS.  The  Navy,  recognizing  the  importance  of  managerial  input 
into  training  decisions,  developed  DOTS  (Design  of  Training  Systems) 
(Duffy  and  Stanley,  1976).  DOTS'  objective  is  to  provide  training 
management  with  computerized  math  models  to  assist  in  predicting  quan¬ 
titative  impacts  of  training  decisions.  DOTS  is  based  upon  the  input 
of  three  previously  existing  data  base  generation  programs  already  de¬ 
veloped  by  IBM  for  the  Navy.  The  first  includes  the  training  system 
capabilities,  requirements,  and  resources.  The  second  evaluates  the 
level  of  educational  technology.  The  third  generates  the  flow  of  the 
training  process.  These  systems  are  analogous  to  tasks  performed  by 
the  early  stages  of  the  MODIA  user  interface  and  its  resource  utili¬ 
zation  model.  DOTS  however  is  much  more  sophisticated.  Unfortunately, 
it  also  is  very  specific  to  the  Navy  command  structures  for  training 
and  as  such  does  not  appear  to  have  an  easy  application  to  Army  prob¬ 
lems.  The  three  DOTS  models  feed  a  manager  model  which  is  called  the 
training  analysis  model  (TRAM) .  For  the  Army,  the  greatest  benefit  of 
TRAM  appears  to  lie  in  evaluative  studies  performed  on  the  TRAM  con¬ 
cepts  during  concept  validation.  The  results  of  these  studies  show 
what  areas  in  developing  systems  are  most  likely  to  be  affected  by 
managerial  needs.  There  are  five  categories  which  have  been  derived 
from  this  analysis.  They  are 

1.  The  planning  of  requirements  including:  (a)  the  effect  of 
increments  or  decrements  in  resources;  (b)  the  setting  of 
student  quotas  and  their  integration  into  existing  training 
settings;  (c)  the  analysis  of  attrition  effects;  and  (d)  the 
analysis  of  new  program  impacts. 

2.  The  determination  of  training  rate  feasibilities  including 
current  manpower  needs  and  demands,  and  equipment  and  space 
limitations. 

3.  The  identification  of  possible  areas  including:  (a)  course 
mixes;  (b)  elimination  of  topic  areas;  (c)  reductions  in 
course  length;  and  (d)  trade-off  analysis. 

4.  The  maximization  of  existing  programs  especially  as  they 
effect  the  cross  utilization  of  instructors. 


5.  Personnel  requirements  as  they  impact  on  support  needs  and 
the  flexibility  of  program  structuring. 

Future  Research  Implications  SNAPS/DOTS.  This  area  represents 
one  of  the  most  challenging  topics  for  the  Army.  Development  will  have 
to  be  carefully  coordinated  with  the  current  management  structure  and 
the  format  which  is  finally  designed  for  early  program  development. 
Existing  efforts  appear  to  be  highly  specialized  and  are  not  readily 
applicable  to  early  Army  training  assessment.  Existing  research  ap¬ 
pears  to  have  its  greatest  value  in  identifying  problem  areas  rather 
than  in  presenting  adaptable  techniques  which  can  be  used.  DOTS  il¬ 
lustrates  the  high  level  of  complexity  that  can  be  developed  if  a  care¬ 
ful  study  of  the  potential  user  needs  is  not  made.  The  Army  would  want 
to  move  carefully  in  this  area  until  the  benefits  can  be  seen  to  clearly 
outweigh  the  development  costs. 


Effectiveness  Estimation 


Directly  associated  with  design  trade-off  and  managerial  decision 
is  the  concept  of  overall  system  effectiveness.  In  the  first  sections 
of  this  paper,  the  design  concept  was  directly  related  to  the  ability 
of  the  system  to  meet  a  potential  system  threat.  To  evaluate  whether 
or  not  a  system  can  meet  that  threat,  it  is  necessary  to  produce  some 
method  through  which  total  system  effectiveness  can  be  assessed.  Cer¬ 
tain  requirements  must  be  met  for  such  an  assessment  to  take  place. 

First,  the  system  environment  or  battle  scenario  within  which  the  action 
must  take  place  needs  to  be  determined.  This  is  in  large  part  a  func¬ 
tion  of  threat  and  mission  definition  which  drives  the  system  concept 
development.  Due  to  the  complexity  of  a  modern  weapon  system,  generally 
two  types  of  effectiveness  evaluations  have  taken  place.  The  first 
consists  of  actual  field  measures  taken  at  operational  tests.  The 
second  includes  reduced  measures  based  on  limited  populations  present 
during  breadboard  or  brassboard  system  configurations.  Neither  is 
applicable  for  systems  still  in  early  conceptual  stages.  About  the 
only  effectiveness  estimation  techniques  which  can  be  used  at  this 
stage  are  forms  of  simulation.  Unfortunately,  most  simulation  efforts 
have  used  languages  which  are  designed  primarily  for  hardware  systems 
such  as  ACSL  (Advanced  Continuous  Simulation  Language)  (Manual,  Mitchell 
and  Gauthier  Assoc.,  Inc.,  1975).  This  simulation  procedure  makes  use  of 
differential  equations  and  highly  predictable  hardware-related  proper¬ 
ties  such  as  component  failure  distributions  in  order  to  predict  total 
system  reliability  and  maintainability.  Another  approach  has  been 
FORTRAN  or  JOVIAL  based  specialty  simulations  of  hardware  interactions 
with  environments  such  as  the  ITEM  (Interactive  Tactical  Environment 
Model)  model  for  air  defense  systems.  The  inclusion  of  a  human  operator 
in  such  models  has  been  a  consistent  problem  area.  One  approach  has 
been  to  model  the  human  operator  as  a  piece  of  highly  flexible  equip¬ 
ment,  i.e.,  a  system  controller.  Developed  primarily  by  engineering 
groups,  this  approach  has  sought  to  find  an  optimal  control  model  for 
the  human  operator  using  system  control  theory  as  a  foundation.  The 
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models  are  usually  highly  complex  mathematically  and  would  be  very  dif¬ 
ficult  to  generalize  to  the  environment  of  the  training  manager.  An¬ 
other  more  recent  approach  has  been  to  develop  a  specialized  language 
specifically  designed  for  the  human  factors  researchers  that  uses  task 
information  as  a  basis  for  capturing  the  man/machine  interaction. 

This  approach  appears  to  have  the  greatest  potential  for  early  system 
specifications . 

SAINT.  SAINT  (Systems  Analysis  of  Integrated  Networks  of  Tasks) 
is  both  a  modeling  technique  and  a  computer  language  (Wortman  £  Duket, 
1978) .  Systems  are  represented  as  graphical  networks  of  tasks  within 
which  one  or  more  operators  interact.  Each  task  in  a  system  is  de¬ 
scribed  as  to  how  it  is  related  to  other  tasks.  A  recoded  graphical 
description  is  appended  to  the  SAINT  computer  program  for  an  automated 
performance  assessment.  The  simulation  includes  both  probabilistic 
and  conditional  task  performance  descriptions  and  precedence  relation¬ 
ships  as  well  as  the  collection  of  statistical  summaries  of  performance 
characteristics.  Besides  possessing  advantages  of  discrete  simulation 
SAINT  can  include  the  full  range  of  continuous  models  considered  with 
ACSL.  A  major  advantage  is  that  SAINT  permits  changes  in  operator 
responses  to  system  internal  or  external  events.  This  makes  SAINT  an 
extremely  good  candidate  for  studying  possible  effects  of  system  trade¬ 
offs  on  overall  system  performance  (Kuperman  et  al.,  1974).  Although 
very  recent  in  origin,  SAINT  is  rapidly  being  recognized  as  a  major 
technique  for  modeling  man/machine  systems.  Of  value  to  this  paper  is 
the  fact  that  the  procedure  has  also  successfully  been  used  within  the 
DAIS  project  to  evaluate  developing  concepts  for  avionics  systems. 

SAINT  would  appear  to  be  a  logical  candidate  for  the  input  from  a  CAPS 
or  THOUGHTSTICKER  developed  concept  mesh.  Such  a  linkup  could  provide 
the  developer  of  both  training  and  materiel  with  unusually  powerful 
tools  for  the  prediction  of  overall  system  effects  and  training  require¬ 
ments  impacts.  ARI  at  Fort  Bliss  already  has  one  ongoing  effort  to 
study  the  prediction  of  training  requirements  based  on  simulated  op¬ 
erator  performance  within  the  AN/TSQ-73  Missile  Minder.  Preliminary 
results  indicate  it  has  a  high  potential  value  for  determination  of 
critical  tasks,  operator  loading  patterns,  and  manning  structures. 


Costing 

Numerous  cost  models  exist  for  the  weapon  life  cycle.  The  pre¬ 
diction  of  cost  at  early  levels  of  system  development  has  been  severely 
limited  by  the  lack  of  precision  in  early  system  concept  formulations 
and  data  bases.  This  area  will  not  be  considered  in  detail  in  this 
paper  other  than  to  state  that  cost  programs  are  being  developed  for 
CTEA  and  COEA  analysis  which  hold  promise  of  being  suitable  for  system 
predictions  as  'well.  Much  of  the  specific  form  of  such  models  (as 
was  the  case  for  Management  Information  Systems)  must  await  further 
development  of  initial  concept  specification  techniques. 
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CONCLUSIONS  AND  FUTURE  DIRECTIONS 


This  paper  has  considered  a  variety  of  developing  and  applied 
techniques  which  have  potential  use  in  the  area  of  early  system  speci¬ 
fication.  Each  of  the  efforts  has  had  certain  strengths  and  weaknesses. 
The  major  emphasis  of  each  is  presented  in  Figure  6.  This  table  by 
no  means  includes  all  related  research.  Nonetheless,  a  pattern  does 
begin  to  emerge  from  which  it  is  possible  to  project  the  basic  ingredi¬ 
ents  for  a  system  that  supplies  assessment  information  to  training 
developers.  Such  a  system  has  been  hinted  at  throughout  the  presenta¬ 
tion  of  techniques  and  problems.  Now  it  will  be  made  specific.  Fig¬ 
ure  7  presents  the  first  approximation  of  what  such  a  system  could 
look  like  based  on  functional  areas. 


A  Tentative  Structure  for  an  Early  Training  Assessment 
System  (ETAS) 


The  system  begins  with  a  perceived  force  deficiency  and  a  speci¬ 
fied  threat  against  which  a  new  weapon  must  be  evaluated.  Step  one 
consists  of  the  production  of  an  initial  hardware  concept.  In  order 
to  make  the  concept  of  value  to  the  largest  potential  audience  stand¬ 
ardization  is  required.  To  standardize  the  concept  form  two  condi¬ 
tions  must  be  met.  First,  developers  must  be  able  to  reproduce  ideas 
in  a  fashion  which  permits  different  users  to  easily  exchange  informa¬ 
tion  and  assess  the  impact  of  different  hardware  possibilities.  Second, 
output  must  be  standardized  in  such  a  way  that  developmental  aids  such 
as  computerized  training  programs  can  accept  the  input  with  minimal 
modifications.  In  addition,  standardized  specifications  are  needed  to 
provide  improved  concept  structures  for  potential  hardware  or  training 
innovators  outside  the  Army  who  would  receive  requests  for  proposals. 

The  first  need  therefore  is  for  a  concept  production  aid  which 
permits  a  creative  individual  to  explicitly  set  forth  and  examine  hard¬ 
ware  and  training  possibilities.  Because  of  the  complexity  of  military 
systems,  the  number  of  alternative  design  configurations  can  expand 
very  quickly  and  often  several  team  members  must  work  together.  Thus, 
the  system  must  also  serve  the  function  of  an  automated  designer's 
notebook  to  permit  additions,  changes,  and  exploratory  efforts  in  a 
standard  communicable  form.  Based  on  earlier  discussions  in  this 
paper,  the  approach  used  by  THOUGHTSTICKER  would  be  an  excellent  po¬ 
tential  candidate  for  just  such  a  system.  Adjustments  would  have  to 
be  made  by  means  of  a  user  program  interface  to  output  both  task  meshes 
and  hardware  information  meshes  simultaneously  to  feed  the  combat  and 
training  development  community.  Such  a  process  does  not  appear  beyond 
the  present  state-of-the-art.  Additional  psychological  research  is 
needed  to  give  precise  specification  of  an  optimal  interface  structure 
between  potential  users  with  different  styles  of  creative  development. 
Such  a  system  would  present  an  excellent  test  bed  for  the  study  of 
creativity  in  system  design  as  well  as  provide  immediate  benefits  for 


Figure  7.  An  early  training  concept  development  system. 


system  specification  within  the  requirements  of  Army  Reg.  A109  and  the 
LCSMM. 


Once  a  graphic  output  mesh  had  been  produced  that  reflected  hard¬ 
ware  and  human  behaviors  within  a  given  mission,  the  next  step  could 
begin.  Since  initial  hardware  and  training  concepts  must  be  evaluated, 
the  initial  mesh  of  objects  and  operator  actions  would  be  programmed 
through  an  automated  simulation  interface  such  as  CAPS.  The  use  of 
CAPS  would  allow  the  designer  to  study  the  total  system  effect  of  sub¬ 
unit  changes.  This  would  produce  early  estimates  of  resource  impacts, 
design  flaws,  and  operator  demands.  Depending  upon  the  level  of  speci¬ 
fication,  SAINT  might  be  applied  to  produce  automatic  tracking  of  im¬ 
pacts  of  various  task  structures  on  the- operators.  This  would  be  of 
value  in  identifying  high  risk  tasks,  true  task  criticality,  and  oper¬ 
ator  overloads  within  an  anticipated  combat  environment.  Modification 
to  the  initial  SAINT  language  may  be  needed  to  simplify  the  coding  of 
THOUGHTSTICKER  developed  task  networks  into  a  form  usable  by  SAINT 
subroutines.  Valuable  research  insights  into  critical  components  of 
a  task  description  format  can  be  gained  by  such  an  analysis.  This  is 
particularly  true  for  determination  of  later  training  program  devel¬ 
opment  needs. 

A  second  output  from  the  graphic  meshes  would  be  input  for  a  SAT 
training  development  approach.  This  method  appears  so  well  designed 
that  an  interface  appears  to  be  relatively  straightforward.  Adaptation 
of  the  SAT  to  Army  uses  appears  to  be  a  very  viable  possibility.  Such 
a  process  could  be  of  direct  value  in  terms  of  early  generation  of 
training  programs. 

An  initial  training  program  format  combined  with  early  simula¬ 
tions  of  the  hardware  and  operator  subsystems  would  in  turn  provide 
input  for  a  training  manager  such  as  the  TSM.  The  input  would  reflect 
the  requirements  of  COEA  and  CTEA  documents  needed  at  milestone  one. 

An  additional  advantage  of  the  approach  is  that  all  decisions  would 
be  documented  for  both  decisionmakers  and  future  development  efforts. 
This  is  especially  true  for  those  efforts  requiring  historical  infor¬ 
mation  to  study  interrelationships  between  various  system  concepts 
and  the  actual  later  field  performance  estimates.  This  is  critical 
should  major  battlefield  simulation  efforts  be  heavily  used  in  assess¬ 
ment  of  field  effectiveness. 


Suggestions  for  Action 

In  terms  of  actions  which  can  be  taken  as  a  result  of  this  exami¬ 
nation,  there  are  four  which  appear  productive.  First,  the  researchers 
and  developers  associated  with  THOUGHTSTICKER,  SAINT,  SAT,  and  CAPS 
should  be  contacted  to  determine  what  advances,  if  any,  are  projected 
in  the  current  state  of  each  effort.  Second,  a  contractual  effort 
should  take  place  to  examine  in  detail  agency  implications  and  per¬ 
ceived  benefits  of  such  a  system  for  the  entire  life  cvcle  of  a  given 


weapon  system.  Third,  in-house  efforts  should -be  undertaken  to  exam¬ 
ine  the  feasibility  of  immediate  application  of  SAT  to  the  Army  train¬ 
ing  development  process.  This  is  particularly  true  regarding  the 
possible  use  of  SAT  in  the  ISD  process.  Finally,  the  utilization  of 
CAPS  should  be  considered  across  a  wider  range  of  Army  problems.  The 
ability  to  generate  rapid  on-line  system  models  with  minimal  user 
training  has  great  potential  payoff  in  addressing  a  whole  class  of 
problems  which  previously  had  been  important  but  of  too  short  a  time 
frame  for  professional  simulation  evaluations.  CAPS  may  permit  a  user 
to  directly  answer  the  problem  without  having  to  go  through  outside 
agencies . 


ABBREVIATIONS 


ACSL 

CAPS 

COEA 

CODAP 

CTEA 

DARCOM 

DOTS 

ECSL 

GANTT 

HQDA 

ILIR 

ISD 

ITEM 

LCSMM 

LSAR 

MENS 

MOS 

MODIA 

OACSI 

OJT 

OMB 

PERT/CPM 

PQQPRI 

SAT 

SNAP 

STEPS 

STOG 

TECEP 

TDDA 

TDIS 

TEEM 

TRADOC 

TRAINVICE 

TRAM 

TSM 


Advanced  Continuous  Simulation  Language 

Computer  Aided  Programming  System 

Cost  and  Operational  Effectiveness  Analysis 

Comprehensive  Occupational  Data  Analysis  Programs 

Cost  Training  Effectiveness  Analysis 

Development  and  Readiness  Command 

Design  of  Training  Systems 

Extended  Control  and  Simulation  Language 

Charting  Process  Invented  by  H.  L.  Gantt 

Headquarters  Department  of  the  Army 

Independent  Laboratory  In-House  Research 

Instructional  Systems  Development 

Interactive  Tactical  Environment  Model 

Life  Cycle  System  Management  Model 

Logistics  Support  Analysis  Record 

Missions  Element  Needs  Statement 

Military  Occupation  Specialty 

Method  of  Designing  Instruction  Alternatives 

Office  of  the  Assistant  Chief  of  Staff  for  Intelligence 

On-the-Job  Training 

Office  of  Management  and  Budget 

Program  Evaluation  and  Review  Technique/Critical  Path  Method 
Provisional  Qualitative  Quantitative  Personnel  Requirements 
Information 

Systems  Analysis  of  Training 
Simplified  Network  Analysis  Portrayal 
Simulation  and  Training  Equipment  Sources 
Science  and  Technology  Operations  Guide 
Training  Evaluation  Cost  Evaluation  Program 
Training  Developers  Decision  Aid 
Training  Developments  Information  System 
Training  Efficiency  Estimation  Model 
Training  and  Doctrine  Command 
Training  Device  Effectiveness  Model 
Training  Analysis  Model 
TRADOC  System  Manager 
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